Papers with Transformer-based pre-trained language models
Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing studies have shown that larger pre-trained language models with more parameters and lower perplexity are less predictive of human reading times. |
| Approach: | They propose to use a transformer-based model with more parameters and lower perplexity to investigate why these models are less predictive of human reading times. |
| Outcome: | The results show that the larger models with more parameters and lower perplexity are less predictive of human reading times and eye-gaze durations collected during naturalistic reading. |
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing approaches to analyze syntactic knowledge of pre-trained language models have been limited. |
| Approach: | They propose an unsupervised method that extracts constituency trees from PLM attention heads. |
| Outcome: | The proposed method outperforms existing approaches if no development set is present. |
COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing statically compressed pre-trained language models lack spatial and temporal efficiency due to their large size and wide width. |
| Approach: | They propose a spatially and temporally efficient model which retains the major capacity of PLMs. |
| Outcome: | The proposed model retains the major capacity of pre-trained language models at high compression and acceleration rate with 1/8 parameters and 1/19 FLOPs of BERT. |
Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing language models such as Transformer-based models fail to predict the conversation outcome. |
| Approach: | They propose to integrate human-annotated domain knowledge and LLM-generated features to provide richer context to counseling conversations. |
| Outcome: | The proposed model improves by 15% when combined with human-annotated domain knowledge and LLM-generated features. |
DABERT: Dual Attention Enhanced BERT for Semantic Matching (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models for semantic sentence matching lack the ability to capture subtle differences. |
| Approach: | They propose to use a Transformer-based pre-trained language model to capture fine-grained differences in sentence pairs by introducing a dual attention module and a fusion module to learn the aggregation of difference and affinity features. |
| Outcome: | The proposed method is able to capture fine-grained differences in sentence pairs. |
Better Pre-Training by Reducing Representation Confusion (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-trained language models address information confusion in position encoding and model representations. |
| Approach: | They propose two techniques to improve pre-trained language models by decoupling directions and auxiliary regularizers. |
| Outcome: | The proposed techniques can improve pre-trained language models on GLUE benchmarks. |
SideControl: Controlled Open-domain Dialogue Generation via Additive Side Networks (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate pre-trained language models with attributes are expensive and overfitted on small training sets. |
| Approach: | They propose a novel approach to control the generation of Transformer-based pre-trained language models using a new control attributes loss framework. |
| Outcome: | The proposed method is shown to perform well with very limited training samples. |
Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pruning results on benchmark transformers, such as BERT, are not as remarkable as those of convolutional neural networks. |
| Approach: | They propose to apply a knowledge-aware pruning process to transformer-based pre-trained language models to reduce model size and model weight. |
| Outcome: | The proposed pruning method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy. |
Nibbling at the Hard Core of Word Sense Disambiguation (2022.acl-long)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a task that is based on a set of pre-trained language models. |
| Approach: | They propose to use Word Sense Disambiguation to test whether systems can handle ambiguous words. |
| Outcome: | The proposed benchmarks show that seven of the most representative state-of-the-art systems make trivial errors on traditional evaluation benchmarks. |
GhostBERT: Generate More Features with Cheap Operations for BERT (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that some parameters in pre-trained language models can be pruned away without severe accuracy degradation. |
| Approach: | They propose a method which generates more features with very cheap operations from the remaining features and can be applied to unpruned BERT models to enhance their performance. |
| Outcome: | Empirical results on the GLUE benchmark on three backbone models (i.e., BERT, RoBERTa and ELECTRA) verify the efficacy of the proposed method. |
HashFormers: Towards Vocabulary-independent Pre-trained Transformers (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. |
| Approach: | They propose a family of vocabulary-independent pre-trained transformers that support unlimited vocabulary . they propose to map each token to its corresponding embedding by default . |
| Outcome: | The proposed models are more memory efficient than existing models while achieving comparable performance on multiple text classification tasks. |
GIFT: Graph-Induced Fine-Tuning for Multi-Party Conversation Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods on understanding multi-party conversations typically embed interlocutors and utterances into sequential information flows or use superficial graph structures. |
| Approach: | They propose a plug-and-play method which adapts Transformer-based pre-trained language models for universal MPC understanding. |
| Outcome: | The proposed method can adapt Transformer-based pre-trained language models for universal MPC understanding. |
Deciphering Stereotypes in Pre-Trained Language Models (2023.emnlp-main)
Copied to clipboard
Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, Andrew Koulogeorge, Lili Wang, Diyi Yang, Soroush Vosoughi
| Challenge: | Current approaches for examining stereotypes in PLMs require intricate human knowledge about these stereotypes and entail careful manual curation of examples. |
| Approach: | They propose a framework for examining stereotype-encoding behavior of PLMs using model probing and textual analyses. |
| Outcome: | The proposed approach can debiase PLMs without compromising their language modeling capabilities or performance. |
Finding Skill Neurons in Pre-trained Transformer-based Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have demonstrated superior performance on various natural language processing tasks. |
| Approach: | They find that after prompt tuning, some neurons encode task-specific skills . they also show that skill neurons are most likely generated in pre-training . |
| Outcome: | The neurons are highly predictive of task labels after prompt tuning for specific tasks. |